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ABSTRACT:  Models  and  simulations  (M&S) 

provide  an  analytical  representation  of  complex 
systems  that  can  be  used  in  the  evaluation  of  those 
systems  in  their  operating  environment  M&S, 
combined  with  other  test  methods  such  as  Hardware- 
in-the-Loop  Tests  (HWILTs)  and  live  test  events,  form 
a  very  powerful  and  valuable  tool  for  the  test  and 
evaluation  of  complex  systems.  The  incorporation  of 
M&S  allows  the  test  program  to  be  better  focused  on 
the  critical  evaluation  areas.  The  early  use  of  M&S 
in  the  system  analysis,  requirements  definition,  as 
well  as  system  test  and  evaluation  provides  for  a 
more  comprehensive  system  engineering  process. 
This  paper  presents  a  strategy  for  the  integration  and 
use  of  models  and  simulations  (M&S)  in  a  complex 
Battle  Management  Command,  Control  and 
Communications  (BM/C3)  Test  and  Evaluation 
(T&E)  Program.  M&S  are  needed  to  assess  the 
contribution  of  BM/C3  to  the  interoperability  of 
various  missile  defense  systems  operating  in  a  Family 
of  Systems  (FoS)  context  in  a  theater  of  operations. 
Models  and  simulations  will  be  incorporated  into  the 
TMD  FoS  T&E  program  using  the  principles  of  the 
Simulation  Test  and  Evaluation  Process  (STEP).  FoS 
T&E  will  be  conducted  to  demonstrate  program 


development  and  system  performance  throughout  the 
life  cycle  of  the  FoS.  Analysis  will  be  the  primary 
evaluation  tool  in  the  early  phase  of  each  increment. 
M&S  will  be  used  in  pre-test  planning  and  prediction 
as  well  as  post-test  analysis.  FoS  level  exercises  will 
be  conducted  to  assess  the  interoperability  and 
performance  of  the  FoS.  Based  on  data  gathered 
from  the  live  events,  M&S  will  be  used  to  assist  in 
model  V&  V  activities.  This  model-test-model 
paradigm  will  be  repeated  throughout  the 
development  and  evaluation  efforts. 

1.  INTRODUCTION 

This  paper  presents  a  strategy  for  the  integration  and 
use  of  models  and  simulations  (M&S)  in  the  Theater 
Missile  Defense  (TMD)  Family  of  Systems  (FoS) 
Test  and  Evaluation  (T&E)  Program.  M&S  are 
needed  to  assess  the  interoperability  of  the  various 
TMD  weapons  systems  and  Command  and  Control 
(C2)  centers  operating  in  a  FoS  context  throughout  a 
theater  of  operations. 

T&E  is  the  FoS  assessment  process  that  provides  the 
necessary  information  to  demonstrate  an  integrated 
FoS  capability,  identify  and  define  interoperability 
problems,  assess  FoS  status,  and  obtain  data  to 
validate  and  refine  the  models  and  simulations. 
Models  and  simulations  provide  an  analytical 
representation  of  complex  systems  that  can  be  used  in 
the  evaluation  of  TMD  systems  in  their  operating 
environment.  M&S,  combined  with  other  test 
methods  such  as  Hardware-in-the-Loop  Tests 
(HWILTs)  and  live  test  events,  form  a  powerful  tool 
for  the  test  and  evaluation  of  complex  systems. 

M&S  will  be  incorporated  into  the  T&E  program  to 
provide  the  opportunity  to  aid  in  event  planning, 
assess  those  objectives  that  cannot  be  tested  in  a  live 
environment,  minimize  or  reduce  risk  of  live  test 
events,  and  increase  the  tester’s  statistical  confidence 
level.  Results  obtained  from  HWILTs  and  live  test 
events  can  then  used  to  validate  the  models.  This 
provides  a  cost  effective  way  to  verify 
interoperability  of  Service  systems  before  they  are 
fielded  to  operational  units.  The  use  of  M&S  in  FoS 
T&E  will  be  in  three  basic  areas:  pre-test  planning 
and  prediction,  post-test  validation  activities,  and 
Theater-level  analysis.  The  primary  simulations  for 
use  in  the  FoS  T&E  program  will  be  the  Extended 
Air  Defense  Testbed  (EADTB)  and  the  Extended  Air 
Defense  Simulation  (EADSIM). 

1.1  TMD  Family  of  Systems  IFoS) 

The  TMD  FoS  consists  of  Service  weapon  systems 
and  C2  centers  that  are  defined  in  the  BMDO’s  core 


1 

BTIC  QUALITY  IwaPBOTBD  4  UNCLASSIFIED 

Approved  for  Public  Release;  Distribution  is  unlimited. 


19991116  013 


UNCLASSIFIED 


program.  Also  included  are  upgrades  to  Service 
weapon  systems  and  C2  centers.  TMD  FoS  testing  is 
being  conducted  by  the  TMD  Systems  Engineering 
and  Integration  (SE&I)  Contractor  Team. 

1.2  FoS  T&E  Methods 

The  FoS  T&E  program  is  designed  to  build  on  the 
individual  Service  development  programs  to  include 
Service  T&E  activities,  as  shown  in  Figure  1-1.  The 
purpose  of  joint  T&E  is  to  focus  on  requirements 
verification  to  ensure  the  successful  integration  and 
interoperability  of  the  independently  developed  TMD 
C2  centers,  sensors,  and  weapon  systems  into  a 
cohesive  FoS.  The  FoS  T&E  program  is  predicated 
on  three  test  methods:  live  test  events;  HWILTs, 
which  involve  the  actual  tactical  systems;  and  Models 
and  Simulations. 


Figure  1-1.  TMD  T&E  FoS  Test  Methods 
The  wedge  in  Figure  1-2  indicates  that  as  fidelity 
increases,  the  number  of  tests  or  trials  that  can  be  run 
decreases.  Live  fire  test  events  provide  the  highest 
level  of  fidelity,  but  due  to  their  high  cost,  very  few 
events  can  be  run.  Live  events  also  lack  repeatable 
results  due  to  the  large  number  of  testing  variables. 
FoS  T&E  relies  on  all  three  methods  with  data  being 
exchanged  between  them.  The  FoS  T&E  strategy  will 
employ  the  “model-test-model”  paradigm  for  better 
prediction  and  M&S  assessment  capability.  Through 
this  approach,  M&S  tools  will  first  be  used  in  test 
planning  and  prediction,  then  the  test  event  will  be 
executed,  and  finally,  test  event  data  is  used  for  M&S 
validation  and  improvement. 

While  M&S  provides  the  community  the  ability  to 
control  and  test  to  significantly  varying  test 
conditions,  a  question  of  confidence  in  the  results 
will  always  remain.  A  bridge  between  M&S  and  live 
tests  is  the  HWILT.  The  Theater  Missile  Defense 
System  Exerciser  (TMDSE)  provides  the  ability  to 
test  tactical  hardware  with  simulated  threats  to 


evaluate  the  FoS  during  the  Service  systems’ 
development.  “The  TMDSE  is  a  real-time, 
geographically  distributed,  hardware-in-the-loop  test 
capability  that  stimulates  actual  tactical  missile 
systems  and  early  warning  systems”*^.  HWILT 
events  can  test  tactical  hardware  with  a  variety  of 
threats  in  any  simulated  theater. 

In  the  conduct  of  FoS  T&E,  both  M&S  and  HWILTs 
are  used  for  risk  mitigation  of  live  test  events  by 
providing  a  characterization  of  the  events  expected 
during  the  live  test.  A  major  objective  of  all  live 
testing  is  to  collect  data  that  can  be  used  to  validate 
both  M&S  and  HWILT  tools  and  to  support  the 
accreditation  of  these  tools.  In  addition,  both  M&S 
and  HWILTs  provide  the  opportunity  to  extend  live 
test  results  to  assess  FoS  performance  in  scenarios 
and  environments  that  are  not  possible  during  live 
testing. 

1.3  Characteristics  of  Models  and  Simulations 
M&S  are  used  in  FoS  assessments  for  few-on  few  to 
many-on-many  engagements  and  large  theater 
architectures.  M&S  are  the  only  tools  that  can  be 
used  to  examine  Theater-wide  FoS  issues  in  scenarios 
that  include  a  realistic  number  of  threats  expected 
under  tactical  conditions  (including  stressing 
environments).  This  realistic  representation  of  the 
threat  is  based  on  the  BMDO  Directorate  of  Threats 
and  Countermeasures  (BMDO/DTC)  approved  data. 

Key  M&S  characteristics  include  the: 

a)  Ability  to  represent  a  variety  of  weapons  and  radar 
systems  at  a  high  level  of  fidelity 

b)  Availability  of  a  wide  range  of  high  fidelity  threats 
and  environments 

c)  Ability  to  examine  performance  under  a  large 
number  of  threats  and  a  heavily  loaded 
communication  networks  at  a  high  level  of  fidelity 

d)  Ability  to  assess  TMD  systems  in  Theater-level 
tactical  deployments 

e)  Use  of  simulated  TMD  systems  and  supporting 
communication  networks 

f)  Ability  to  create  a  scenario  anywhere  in  the  world 

There  are  several  advantages  and  limitations  to  using 
models  and  simulations: 

Advantages: 

•  Permits  controlled  experimentation  and 
repeatability  of  experiment  conditions 

•  Permits  time  compression 
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•  Permits  sensitivity  analysis  by  manipulation  of 
input  variables 

•  Does  not  disturb  the  real  system. 

•  Ability  to  gain  statistical  confidence  through 
Monte  Carlo  runs 

•  Lowest  cost  method  of  collecting  data 
Limitations: 

•  Availability  and  accuracy  of  platform  performance 
characteristics. 

•  Cannot  account  for  the  decisions  made  by  the 
tactical  system  operators  in  a  live  tactical 
environment 

•  Simple  human  decision  modeling 

•  Limited  operator  involvement 

•  Extensive  set-up  time  may  be  encountered  and 
model  parameters  may  be  difficult  to  initialize. 
These  may  require  extensive  time  in  collection, 
analysis,  and  interpretation. 

•  Level  of  validation  of  the  M&S  using  real-world 
data  and  subsequent  user  accreditation. 

•  Lower  confidence  results 

Given  the  aforementioned  advantages  and 
limitations,  M&S  is  a  vital  part  of  a  complete  T&E 
program  when  used  in  conjunction  with  HWILTs  and 
live  events.  Incorporating  M&S  into  the  T&E 
program  allows  the  test  program  to  be  better  focused 
on  the  critical  areas  for  evaluation.  The  early  use  of 
M&S  in  the  system  analysis,  definition  of 
requirements,  and  system  test  and  evaluation 
provides  for  a  more  comprehensive  system 
engineering  process. 


2.  STRATEGY  FOR  INTEGRATING  M&S 
INTO  FoS  T&E 

Models  and  simulations  will  be  incorporated  into  the 
TMD  FoS  T&E  program  using  the  principles  of  the 
Simulation,  Test  and  Evaluation  Process  (STEP)®  and 
will  conform  to  DoD  Directive  5000.59-P. 

2.1  Incorporation  Of  M&S  Into  FoS  T&E 
FoS  T&E  will  be  conducted  to  demonstrate  program 
development  and  system  performance  throughout  the 
life  cycle  of  the  FoS.  FoS  capability  will  evolve 
through  a  series  of  increments,  with  each  increment 
serving  as  the  basis  for  the  next.  Analysis  will  be  the 
primary  evaluation  tool  in  the  early  phase  of  each 
increment.  Simulations  will  be  used  in  the  analysis 
to  evaluate  design  tradeoffs  and  candidate  models 
will  be  identified  for  use  in  further  analysis.  M&S 
will  be  used  in  pre-test  planning  and  prediction  as 
well  as  post-test  analysis.  FoS  level  exercises  will  be 
conducted  to  assess  the  interoperability  and 
performance  of  the  FoS.  Based  on  data  gathered  from 
the  live  events,  M&S  will  be  used  to  assist  in  model 
V&V  activities.  This  paradigm  will  be  repeated 
throughout  the  development  efforts.  This  is  the 
“model-test-model”  approach  that  has  become  STEP. 

The  STEP  approach  integrates  the  use  of  M&S  with 
test  events,  each  depending  on  the  other,  to  add  value 
to  the  T&E  process.  M&S  are  part  of  a  strategy  to 
provide  information  by  predicting  test  results,  by 
exploring  performance  in  the  testable  realm,  and  by 
extrapolating  performance  in  the  non-testable  realm. 
Tests  are  used  to  provide  data  from  the  real  world 
regarding  situations  and  environments  and  permit  an 
assessment  of  the  attainment  of  technical 
performance  specifications  and  system  maturity®. 
Figure  2-1  depicts  an  example  of  how  M&S  will  be 
integrated  into  live  FoS  T&E  events  and  illustrates 
the  model-test-model  approach  being  used. 
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Figure  2-1 .  M&S  Coupled  to  Live  Test  Events. 


In  this  example,  input  data  needed  to  accurately 
represent  the  systems  in  EADTB,  EADSIM  and 
TMDSE  will  be  identified  in  pre-event  modeling 
activities.  This  data  will  be  identified  for  collection 
during  the  live  event  and  will  be  used  in  post-event 
simulation  runs  to  assist  with  model  validation  and 
with  post-test  analysis.  Data  collected  from  the  event 
will  be  used  to  satisfy  the  allocated  Measures  Of 
Performance  (MOPs). 

M&S  will  be  used  in  three  major  areas  for  the  FoS 
T&E  program:  pre-test  planning  and  prediction, 
post-test  analysis  and  validation  activities,  and 
theater-level  FoS  analysis  and  assessment.  The  near- 
term  focus  for  T&E  M&S  events  will  be  to  establish 
the  necessary  procedures  and  coordination  needed  to 
effectively  use  the  M&S  tools  to  address  FoS  T&E 
requirements.  A  primary  activity  associated  with  the 
near-term  use  of  M&S  will  be  to  use  data  from  other 
T&E  events  to  begin  validation  of  the  M&S  tools  that 
will  support  the  FoS  T&E  program. 

The  primary  uses  for  M&S  will  be: 

a.  Test  Planning  and  Prediction 

•  Support  scenario  selection  and  development 

•  Prediction  of  test  results 

•  Confirmation  of  data  collection  plans  and  dry 
run  data  analysis  plans 

•  Identify  additional  validation  data  for  collection 

b.  Post-event  M&S  activities 


•  Compliment  HWILTs,  System  Integration  Tests 
(SITs)  and  other  live  events 

•  Compare  M&S  results  with  test  results 

•  Analyze  test  anomalies 

•  Assist  with  M&S  validation  activities 

c.  Theater-level  performance  of  the  FoS  against 
large  numbers  of  threats 

•  many-on-many  scenarios  in  a  variety  of  theaters 

•  FoS  performance  under  peak  network  loading 

•  network  performance  evaluation  under  full 
loading  conditions 

•  Performance  of  the  FoS  versus  the  same  systems 
operating  autonomously 

•  Battle  Management/Command,  Control  and 
Communications  (BM/C3)  effectiveness  at  the 
theater  and  control  levels 

•  Evaluation  of  timelines  required  in  information 
flow  for  cueing,  engagement  coordination,  kill 
assessment,  etc. 

•  Extension  of  live  test  activities  (number  of 
threats/different  environments) 

2.2  General  M&S  Strategy 
The  general  process  for  using  M&S  in  the  FoS  T&E 
program  is  illustrated  in  Figure  2-2.  This  process 
consists  of:  1)  evaluating  and  selecting  M&S  tools 
based  on  the  T&E  requirements,  2)  using  the  M&S 
tools  in  support  of  T&E  events,  3)  using  data  from 
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T&E  events  to  support  V&V  activities  and  4)  to 

provide  feedback  to  the  test  planning  process.  The 

steps  required  to  execute  this  process  are  listed 

below. 

1.  Evaluation  and  selection  of  appropriate  M&S 
tools  based  on  requirements.  This  process  will 
continue  throughout  the  life  of  the  T&E  program 
as  the  current  M&S  tools  evolve  and  as  new 
tools  become  available. 

2.  Establish  the  hierarchical  relationship  between 
the  chosen  M&S  tools. 

3.  Establish  the  linkage  between  the  chosen  M&S 
tools  and  the  T&E  requirements/objectives. 
MOPs  will  be  allocated  to  M&S  events  and 
M&S  tools  will  be  chosen  that  can  provide  the 
data  to  support  the  assessment  of  those  MOPs. 

4.  Tie  into  and  leverage  off  analysis  results. 
Through  this  process,  T&E  is  involved  in  the 
requirements  definition  phases  and  can  utilize 
the  results  to  gain  a  full  understanding  of  the 
system  requirements  and  gain  insight  into  the 
performance  and  utilization  of  the  M&S  tools. 
This  will  aid  in  the  development  of  the  model 
hierarchy,  the  evaluation  of  the  tools,  the  types 
and  content  of  tool  input  and  output  data, 
analysis  of  the  output  data,  and  suggestions  for 
improvements  in  the  tools. 

5.  Verification  and  calibration  of  the  selected  M&S 
tools  through  the  use  of  data  from  live  T&E 
events  as  well  as  data  from  MDAP  models.  This 
process  is  necessary  to  gain  confidence  in  the 
tools  and  to  ensure  continuous  improvements  in 
the  models. 

6.  Use  of  M&S  in  pre-test  activities,  post-test 
activities,  and  Theater-level  analysis. 

7.  Use  of  data  and  results  from  M&S  and  other 
T&E  events  to  revise  test  plans  and  requirements 
and  to  provide  feedback  to  model  developers. 

8.  Population  of  a  data  repository  to  provide  the 
community  with  timely,  verified,  and  validated 
data  for  use  in  other  programs. 

9.  Creation  and  population  of  data  commonality 
matrices  for  the  purposes  of  maintaining  a  record 
of  data  that  is  common  between  the  various 
M&S  tools  and  live  events. 


Figure  2-2.  TMD  FoS  T&E  M&S  Process. 

Through  the  implementation  of  the  above  processes, 
the  use  of  M&S  in  the  FoS  T&E  program  becomes 
one  piece  of  a  larger  process  in  which  M&S  is  used 
throughout  the  systems  engineering  cycle.  The 
coordinated  use  of  M&S  in  the  areas  of  analysis, 
engineering,  requirements  definition,  as  well  as 
system  test  and  evaluation  provides  a  more  robust 
and  efficient  process  for  the  development  and 
evaluation  of  the  TMD  Family  of  Systems.  The  most 
important  components  of  this  system  engineering 
approach  are  a  detailed  plan  for  the  use  of  M&S 
throughout  the  process,  early  selection  and  evaluation 
of  the  tools  to  be  used,  ensuring  the  credibility  of  the 
tools  to  be  used,  using  the  resulting  data  to  improve 
the  tools,  and  ensuring  the  flow  of  data  to  all  key 
players  in  the  process.  Early  and  iterative  use  of 
M&S  in  the  systems  engineering  process  can  aid  in 
the  identification  of  critical  test  issues  and  the 
allocation  of  MOPs  to  particular  test  events.  M&S 
can  also  be  used  to  design  more  effective  T&E 
events,  extrapolate  the  results  from  live  test  events, 
and  to  better  understand  the  aspects  of  system 
performance  observed  during  test  events.  This 
process  embodies  the  spirit  of  the  STEP  process  as 
well  as  the  intent  of  DoD  5000.2-R^ 

3.  IMPLEMENTATION  OF  THE  M&S 
STRATEGY 

The  implementation  and  application  of  these 
processes  will  integrate  all  applicable  elements  of  the 
STEP  process  into  the  FoS  T&E  program.  The 
implementation  of  this  strategy  will  aid  in  the 
development  of  an  efficient  process  for  the  analysis, 
test  and  evaluation  of  the  FoS. 

3.1  M&S  Tool  Selection  and  Hierarchy 
The  first  step  in  the  M&S  process  is  to  identify  those 
M&S  tools  that  can  be  used  to  satisfy  the  T&E 
objectives  and  requirements  that  are  specified  in  the 
system  requirement  documentation.  Once  the  M&S 
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tools  have  been  identified,  a  hierarchy  must  be 
established.  In  this  hierarchy,  the  relationship  or 
affinity  of  these  models  to  each  other  and  the  flow  or 
exchange  of  data  between  the  tools  is  identified.  The 
M&S  hierarchy  consists  of  four  categories  as  shown 
in  Figure  3-1. 


Figure  3-1.  Modeling  and  Simulation  Hierarchy. 

Engineering/Phenomenology  Models.  These 
models  deal  with  the  actual  physics  involved  with 
specific  system  functions  such  as  radar  cross  section, 
signature,  lethality,  and  debris^ 

Engagement  Models.  These  models  usually  deal 
with  specific  weapons  systems  and  their  components. 
These  high  fidelity  models  are  used  by  the  MDAPS 
and  deal  with  such  issues  as  end-game  lethality^. 

Mission-Level  Models.  These  simulations  are  more 
aggregated  and  may  vary  in  fidelity  from  high  to  low. 
The  simulations  contain  models  of  various  weapons 
systems  and  components  and  can  be  used  to  explore 
few-on-few  to  many-on-many  engagements  between 
a  number  of  systems^. 

Campaign-Level  Models.  These  simulations  are 
usually  highly  aggregated  and  have  low  fidelity 
representations  of  numerous  systems.  They  are 
typically  used  to  provide  insight  into  a  system’s 


contribution  to  force  effectiveness  and  to  study  force 
structures  and  logistical  requirements^. 

The  hierarchy  of  M&S  tools  is  established  to  define 
the  flow  of  data.  Models  at  the  bottom 
(engineering/phenomenology)  provide  data  to  models 
at  the  next  level  (engagement)  which,  in  turn,  provide 
data  for  the  mission  level  models  which  feed  the 
campaign  level  models.  The  models  at  the  lower 
level  address  very  specific  issues.  Each  higher  level 
of  the  hierarchy  addresses  larger  issues,  but  usually  at 
less  fidelity  or  detail.  The  hierarchy  is  used  to 
establish  an  affinity  diagram  such  as  the  one  shown 
in  Figure  3-2.  The  purpose  of  the  affinity  diagram  is 
to  define  how  data  flows  between  M&S  tools  and  test 
events. 

The  legend  for  this  figure  as  well  as  Figure  3-4  is 
shown  in  Figure  3-3.  Figure  3-2  shows  the 
hierarchical  relationship  of  the  models  and 
simulations  used  in  TMD  FoS.  The  diagram  shown 
here  is  notional  and  shown  at  a  low  level  of  detail. 
Since  a  thorough  understanding  of  this  data  flow  is 
critical  to  ensuring  the  success  in  the  incorporation  of 
M&S  into  the  T&E  program,  more  detail  would  need 
to  be  added  to  this  affinity  chart.  Even  at  a  low  level 
of  detail,  however,  the  analyst  can  still  gain  some 
insight  as  to  how  various  M&S  tools  can  be  linked 
together  to  achieve  the  established  T&E  goals  and 
objectives.  This  figure  was  designed  to  provide  an 
overall  view  of  the  simulations  being  used,  or  that 
could  be  used,  and  how  they  relate  to  each  other  as 
well  as  to  FoS  T&E  events.  The  chart  can  be  used  to 
aid  the  analyst  in  choosing  models  for  a  particular 
study  or  T&E  event.  However,  the  more  detail 
contained  in  the  M&S  affinity  diagram  the  more 
useful  it  will  be  in  defining  how  the  M&S  tools  will 
be  applied.  The  T&E  program  will  typically  employ 
models  at  various  levels  of  fidelity  depending  on  the 
issue  being  addressed. 
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It  should  be  noted  that  the  data  flow  between  models 
shown  here  is  in  “general  terms.”  The  specific  data 
that  can  be  utilized  between  models  and  simulations 
is  situation  or  scenario  specific  and  will  change  based 
on  the  types  of  questions  to  be  answered  using  the 
model  or  simulation.  The  purpose  of  showing  the 
“general”  data  flow  is  to  give  the  analyst  an  idea  of 
the  type(s)  of  data  that  each  model/simulation  can 
produce  and  which  models/simulations  can  accept 
that  type  of  data  and  confidence  that  the  results  will 
address  the  questions  with  some  degree  of  fidelity. 


Model/Simulation  used  only  by  TMD  SE&I 

Model/Simulation  used  by  the  Services, 

TMD  SE&I,  and  other  contractors. 

SM6DOF 

j  Model/Simulation  used  only  by  the  Services 

(<  TMUSE 

)  Hardware/Human-in-the-Loop  Tool 

^ - 

Model/Simulation  output  data 

^ ^ 

Data  shared  between  live  tests  and  models/simulations 

< . ► 

Confirmatory  data  or  scenario  data  (scoping) 

Figure  3-3.  Legend  for  Figures  3-2  and  3-4. 


In  Figure  3-4,  the  hierarchy  is  scoped  down  to 
highlight  the  models  and  simulations  primarily  used 
in  TMD  FoS  T&E.  The  figure  shows  how  the  M&S 
tools  relate  to  each  other  as  well  as  how  they  relate  to 
T&E  events.  The  figure  also  depicts  how  data  from 
live  events  is  used  for  validation  of  the  models  and 
where  the  M&S  are  used  for  test  planning  and  risk 
reduction  activities. 
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Figure  3-4.  SE&I  M&S  Relationships  for  TMD  FoS. 


3.2  Linkage  To  FoS  T&E  Requirements 
Test  and  evaluation  requirements  must  be  linked  to 
the  M&S  tools  to  be  used.  The  TMD  FoS  T&E 
requirements  span  three  functional  areas:  Planning 
(JPN),  Coordination  (JDN),  and  Execution  (JCTN). 
M&S  tools  were  selected  based  upon  their  ability  to 
address  issues  in  one  or  more  of  these  functional 
areas.  A  combination  of  tools  is  needed  in  order  to 
span  the  requirements  in  all  three  areas. 

Figure  3-5  shows  the  functional  areas  covered  by  the 
various  test  methods  (Execution,  Coordination  and 


Planning)  as  well  as  the  primary  M&S  tools.  Also 
indicated  in  this  figure  is  the  flow  of  data  that  occurs 
in  order  to  link  the  various  test  methods.  This 
linkage  is  needed  to  provide  for  complete  verification 
of  the  requirements,  which  may  rely  on  more  than 
one  test  method.  Through  this  approach  of  linking 
various  M&S  tools,  HWILTs,  and  live  events,  all 
TMD  FoS  T&E  requirements  can  be  adequately 
addressed. 
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Figure  3-5.  M&S  For  TMD  FoS  T&E. 


3.3  Relationship  Between  The  Analysis  And  T&E 
Functions 

The  use  of  M&S  for  FoS  evaluation  includes  more 
than  T&E  events.  M&S  are  integrated  into  the  T&E 
process  to  support  a  comprehensive  evaluation 
strategy  throughout  the  life-cycle  of  the  system.  The 
use  of  M&S  early  in  the  requirement  definition  phase 
is  important  in  achieving  this  means.  Thus,  the  use 
of  M&S  spans  the  analysis  and  T&E  communities 
and  their  use  should  be  coordinated  and 
complimentary. 

STEP  can  contribute  to  refining  the  TMD  Capstone 
Requirements  Document  (CRD)  requirements.  Using 
M&S  to  support  sensitivity  analyses  can  identify 
performance  drivers  and  aid  in  determining  critical 
technical  and  operational  issues.  Through  analysis, 
M&S  tools  can  be  used  to  identify  and  refine 
Measures  Of  Effectiveness  (MOEs)  and  MOPs  and  to 
establish  objective  and  threshold  criteria  to  be  used  in 
both  the  analysis  and  T&E  efforts®.  M&S  are  critical 
to  the  complete  evaluation  of  the  system  through 
early  analysis  as  well  as  test  and  evaluation.  The  use 
of  M&S  throughout  the  process  of  requirements 
definition,  analysis  and  T&E  provides  the  foundation 
needed  to  establish  a  comprehensive  system 
evaluation  strategy  with  clearly  defined,  measurable 
parameters  with  associated  quantifiable  evaluation 
criteria. 

Figure  3-6  shows  how  the  analysis  and  T&E 
functions  work  together  to  form  a  comprehensive 
evaluation  plan  for  the  TMD  FoS.  Analysis  is  used 


to  transform  the  top  level  requirements  from  the  CRD 
into  system-level  requirements  for  the  SRD.  This 
analysis  also  produces  Technical  Performance 
Measures  (TPMs)  for  evaluation  of  the  system.  The 
SRD  requirements  are  then  verified  through  analysis, 
test  and  evaluation,  or  a  combination  of  analysis  and 
testing  using  the  TPMs.  If  the  requirement  is 
satisfied,  then  the  data  is  stored  in  a  requirements 
verification  matrix,  which  contains  the  requirements 
and  their  traceability  to  the  CRD  as  well  as  the  data 
that  supports  the  satisfaction  of  that  requirement.  If 
the  requirement  is  not  satisfied,  then  it  must  be 
referred  to  further  testing  and/or  analysis.  In  some 
cases,  it  may  be  determined  that  the  requirement 
cannot  be  met.  If  this  happens,  further  analysis  will 
be  required  to  determine  the  effect  of  the  requirement 
not  being  met  on  the  system  as  well  as  the  impact  on 
other  requirements. 

The  ability  to  trace  evaluation  results  to  requirements 
is  vital  to  demonstrating  the  degree  to  which  the 
system  performance  satisfies  the  requirements.  The 
iterative  nature  of  STEP  enables  the  linking  of 
performance  measures  and  derived  parameters  back 
to  the  operational  requirements®. 
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3.4  M&S  Tool  Verification.  Validation  And 
Accreditation  (VV&A) 

One  of  the  most  important  aspects  of  STEP  is  the 
verification,  validation  and  accreditation  (VV&A)  of 
the  M&S  tools  to  be  used. 

The  STEP  approach  involves  integrating  the  data 
from  laboratory  and  field  tests  to  validate  models  and 
simulations  as  the  system  matures®.  This  process  will 
result  in  a  more  robust  set  of  M&S  tools  that  gain 
credibility  as  the  evaluation  program  progresses.  A 
high  degree  of  confidence  and  credibility  in  the 
models  is  required  for  their  use  for  T&E. 

Accreditation  is  usually  the  result  of  a  thorough  V&V 
process.  Although  the  SE&I  Team  does  not  have  the 
responsibility  for  planning  and  conducting  V&V  on 
any  of  the  models  being  used,  both  the  analysis  and 
T&E  efforts  have  a  responsibility  to  ensure  that  the 
M&S  tools  being  used  produce  credible  results  for 
the  intended  uses.  This  responsibility  includes 
tracking  the  results  of  V&V  activities  being 
performed  and  collecting  data  for  comparison  with 
data  from  other  accredited  models  or  applicable  test 
events. 

For  validation  purposes,  a  model  or  simulation  can  be 
partitioned  by  its  inputs,  outputs,  or  by  the  structure 
of  the  model  itself.  Two  widely  accepted  domains 
for  validation  include  “output  validation”  and 
“structural  validation.”*^ 

Output  validation  is  the  most  credible  form  and 
should  be  conducted  at  the  full  model  level  to  the 
extent  possible*®.  When  this  is  not  possible,  the 
model  should  be  decomposed  and  output  validation 
conducted  for  the  components.  Structural  validation 


should  be  accomplished  for  those  aspects  of  the 
model  critical  to  the  model’s  intended  use*®. 

V&V  requires  both  quantitative  and  qualitative 
assessments.  Quantitative  assessments  should  be 
used  to  the  maximum  extent  possible.  Activities  need 
to  be  focused  on  the  specific  areas  or  questions  the 
model  is  going  to  be  used  to  address.  These  activities 
should  provide  for  a  systematic  examination  of  the 
model  and  should  portray  the  capabilities  and 
limitations  of  the  model  in  regard  to  the  questions  the 
model  will  be  used  to  answer.  This  is  particularly 
true  in  the  case  of  the  FoS  program.  Due  to  the 
number  of  models  used  either  directly  or  indirectly 
by  the  SE&I  team,  complete  V&V  of  all  models  will 
not  be  possible.  Most  of  the  models  will  be 
accredited  by  their  respective  development  agencies’ 
use  of  independent  assessors,  possibly  with  the  use  of 
data  obtained  through  the  FoS  T&E  program. 

The  general  process  for  V&V  activities  will  be  to  use 
data  from  live  test  events  and  exercises  for 
comparison  with  M&S  output  data  where  applicable. 
Most  of  the  test  data  will  not  contribute  significantly 
to  validation  of  the  force-on-force  models  with  the 
exception  of  the  EADTB  System  Specific 
Representations  (SSRs).  For  the  most  part,  the  test 
data  will  be  provided  to  the  developers  for  V&V  of 
the  engineering  and  MDAP  models.  Credibility  in 
the  force-on-force  models  will  be  gained  by  using  the 
output  from  the  engineering  and  MDAP  models  as 
input  to  the  force-on-force  models. 

3.4.1  Output  Validation 

In  general,  output  validation  contributes  the  most 
convincing  evidence  in  establishing  the  credibility  of 
a  model*®.  This  process  involves  determining  the 
extent  to  which  the  model  or  simulation  output 
represents  the  significant  features  of  real  world 
systems  and  events.  Output  validation  involves 
collecting  data  from  live  events  such  as  system 
integration  tests  and  joint  exercises  and  comparing 
them  with  the  output  of  the  model*®.  In  the  case  of 
the  MDAP  and  engineering-level  models,  this 
process  will  be  relatively  straight  forward.  During 
the  test  planning  phase,  data  will  be  identified  for 
collection  that  can  be  matched  with  the  output  data 
from  the  various  models.  This  will  include  the 
matching  of  MOEs  and  MOPs  that  can  be  associated 
with  both  test  data  and  model  output. 

For  EADSIM  and  EADTB,  this  process  will  be  more 
difficult.  These  force-on-force  models  involve  a 
complex  interaction  of  systems  and  the  environment. 
Replicating  the  scenario  in  terms  of  the  locations  of 
the  systems  and  the  scripted  events  is  possible,  but 
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replicating  the  interactions  of  the  systems  and  the 
decisions  of  the  human  operators  is  far  more  difficult. 
It  is  also  impossible  to  accurately  model  the  effects 
of  the  environment  on  the  individual  events.  When 
using  live  exercises  for  comparison  purposes,  care 
must  be  taken  to  analyze  only  those  areas  that  can  be 
tightly  controlled,  such  as  the  individual  system 
models,  network  traffic,  etc.  More  generalized 
comparisons  can  be  made  in  terms  of  the  outcomes  of 
engagements,  the  number  of  TBMs  destroyed,  etc. 

In  addition  to  the  quantitative  validation  in  which 
measurable  and  observable  outputs  are  studied, 
qualitative  assessments  are  also  important.  A 
simulation  such  as  EADTB  is  too  large  and  complex 
to  be  verified  solely  by  quantitative  means. 
Therefore,  it  is  imperative  to  rely  on  the  opinions  of 
the  operational  and  technical  subject  matter  experts. 
This  “face  validation”  would  include  obtaining  the 
opinions  of  these  experts  on  the  model  areas  that  are 
of  critical  interest  to  the  FoS.*® 

It  is  extremely  important  that  the  observations  and 
measurements  made  in  the  real  world  are  compatible 
with  the  model  output  and  are  applicable  to  the  areas 
of  interest  the  model  will  be  used  to  study. 

3.4.2  Structural  Validation 

EADTB  is  a  very  complex  simulation  consisting  of 
many  different  models.  Output  validation  of  such  a 
simulation  is  extremely  difficult,  therefore,  structural 
validation  very  important.  Structural  validation  deals 
with  an  a  priori  examination  of  the  model  input  data, 
the  basic  principles  of  the  model,  and  the  model’s 
assumptions  to  determine  the  degree  to  which  they 
are  complete,  logical,  consistent  and  reasonable  for 
the  intended  uses  of  the  modef  Structural  validation 
can  aid  in  developing  confidence  in  the  model  results 
by  demonstrating  the  model’s  internal  integrity. 
Structural  validation  determines  the  degree  to  which 
the  model  input  data  (e.g.,  scenarios,  missions,  and 
data  bases)  and  the  model  (e.g.,  assumptions,  logic, 
and  parameters)  represent  the  real  world. 

For  EADTB,  most  of  this  is  being  accomplished 
through  the  use  of  the  certified  SSRs  in  early  studies. 
Through  these  studies  and  by  working  with  the  SSR 
developers,  a  good  understanding  of  the  model,  its 
data  and  its  assumptions  can  be  gained.  These 
studies  are  conducted  on  very  limited  areas  and  the 
results  are  compared  with  data  from  MDAP  models. 
Military  Standards,  test  data,  and  the  experience  of 
operational  and  technical  subject  matter  experts. 

EADTB  will  be  anchored  to  and  will  receive  data 
from  a  number  of  MDAP  and/or  engineering  level 
models.  Ensuring  the  validation  of  these  feeder 


models  will  also  aid  in  the  credibility  of  EADTB.  A 
majority  of  the  structural  validation  will  be 
accomplished  during  the  early  studies  and  analyses. 

Sensitivity  analysis  is  another  important  aspect  of 
V&V.  Sensitivity  analysis  should  be  conducted  on 
the  model  and  its  decomposed  parts  and  can  be  used 
to  check  for  responses  to  various  inputs  and  to  check 
for  marginal  break  points  and  special  limiting 
values^°.  Sensitivity  analysis  is  useful  in  gaining  a 
better  understanding  of  how  the  model  works  and  to 
help  identify  errors  in  the  model  structure  or  code. 
When  coupled  with  the  comparative  validation 
process,  the  sensitivity  analysis  can  help  ensure  that 
the  models  sensitivities  are  representative  of  the  real 
world.  This  can  be  accomplished  during  the  analysis 
conducted  in  support  of  the  System  Requirements 
Document  (SRD). 

A  thorough  understanding  of  the  model  including  its 
sensitivities  is  very  important  if  the  model  is  to  be 
used  for  T&E.  The  effects  of  various  inputs  on  the 
specific  outputs  of  interest  must  be  explored  and  well 
understood. 

The  most  important  V&V  activity  is  the 
documentation  of  data  and  results.  All  relevant  data 
collected  from  T&E  events  will  archived  be  and  a 
matrix  will  be  developed  that  shows  the  relationship 
between  test  data  and  the  M&S  tools  from  both  an 
input  and  output  perspective. 

3.5  M&S  Implementation  In  TMD  FoS  T&E 
Once  the  M&S  tools  have  been  analyzed  for  their 
contribution  to  FoS  T&E  assessments  and  a  level  of 
confidence  has  been  gained  in  the  data  produced  by 
the  tools,  they  can  be  implemented  into  the  T&E 
effort.  T&E  activities  delegated  to  M&S  tools  include 
pre-  and  post-test  activities  and  theater  level  analysis. 

3.5.1  Pre-Test  Activities 

The  use  of  M&S  in  pre-test  activities  includes 
supporting  the  selection  and  development  of 
scenarios  for  BMDO-controlled  test  events.  These 
events  primarily  include  System  Integration  Tests 
(SITs)  and  HWILTs.  Pre-test  M&S  events  will  have 
the  following  objectives: 

•  ensure  that  the  planned  Red  Force  scenario  and 
Blue  force  laydown  will  meet  the  test 
requirements 

•  ensure  test  objectives  can  be  met 

•  focus  the  test  on  critical  aspects  of  the  test 
objectives 

•  identify  scenarios  that  will  contribute  the  most  to 
the  test  objectives 
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•  determine  the  most  efficient  use  of  test  resources 

•  characterize  the  test  activities  and  predict  test 
results 

•  data  collection  planning,  i.e.,  identify  sources  of 
data  to  support  test  objectives  and  model 
validation  activities 

•  identify  potential  problems  (procedural,  test 
architecture,  data  collection,  data  analysis) 

Most  of  these  objectives  will  be  accomplished 
through  “dry  runs”  of  the  planned  test  event  prior  to 
the  event  in  sufficient  lead-time  to  influence  test 
documentation  with  predicted  results.  Pre-test  M&S 
activities  reduce  the  potential  risks  associated  with 
execution  of  the  test  event  and  provide  confidence  in 
the  achievement  of  test  objectives,  which  is  important 
due  to  the  high  cost  and  in-frequency  of  these  events. 
M&S  could  also  be  used  to  create  scenarios  that  will 
specifically  test  FoS  interoperability  and  performance 
issues.  These  scenarios  can  then  be  utilized  in 
HWILTs  and  other  T&E  events. 

EADSIM  and  EADTB  are  used  for  pre-running 
HWILT  events.  Each  has  the  ability  to  rapidly  model 
all  the  participants  in  the  event  as  well  as  model  the 
region  using  Digital  Terrain  Elevation  Data  (DTED). 
As  EADTB  matures  to  incorporate  a  full  suite  of 
Service  participants  as  well  as  more  defined  threat 
systems,  it  will  become  the  tool  of  choice  for  pre¬ 
running  HWILTs.  Using  M&S  to  model  the  scenario 
allows  the  developer  the  ability  to  change  the 
scenario  based  on  simulated  outcomes.  This  ensures 
that  the  final  scenario  will  test  the  FoS  to  accomplish 
the  required  test  objectives. 

For  live  exercises,  M&S  are  used  to  determine  the 
sequence  of  events  from  launch  to  impact/intercept. 
These  pre-test  simulation  runs  provide  the  ability  to 
mitigate  risk  by  visualizing  the  scenario  prior  to  a 
costly  System  Integration  Test  (SIT)  or  other  live  test 
event.  Live  events  also  provide  a  means  to  obtain 
data  for  use  in  M&S  verification  and  validation 
activities. 

3.5.2  Post-Test  Activities 
Models  and  simulations  benefit  the  tester  by  giving 
him  a  means  with  which  to  identify  and  analyze 
anomalies  that  occurred  during  the  test.  The 
simulations  can  also  be  used  to  play  “what  if’  games 
by  varying  test  parameters  and  comparing  the 
simulation  results  with  the  actual  test  results.  An 
examples  would  be  assessing  the  performance  of  the 
FoS  versus  the  same  systems  operating  autonomously 
or  assisting  in  the  investigation  of  what  may  have 
gone  wrong  in  the  test  and  provide  possible  solutions 
to  be  incorporated  into  the  planning  of  future  tests. 


Post-test  activities  also  includes  the  comparison  of 
the  simulation  outputs  with  the  actual  test  results. 
Post-  event  simulation  runs  are  used  to  compare  the 
simulation  output  data  to  the  actual  event  data.  The 
model  is  then  modified  to  more  accurately  represent 
the  “real  world”.  Test  data  would  be  provided  to  the 
model  developers  for  use  in  model  validation 
activities.  Through  this  process,  testers  gain 
confidence  in  the  M&S  tools  for  their  use  in  future 
test  studies..  Post-test  analysis  will  also  aid  in 
confirming  the  tool’s  ability  to  support  T&E  and  to 
possibly  identify  new  ways  in  which  the  tool  can  be 
used.  This  analysis  will  help  identify  shortcomings 
in  the  model  and  possible  areas  for  enhancement. 
Variations  in  the  results  between  M&S  and  live 
events  will  be  reported  to  BMDO  who  will  feed  the 
information  back  to  the  model  developers.  M&S 
tools  may  also  be  used  to  help  explore  the  cause  of 
differences  between  predicted  and  actual  test  results. 
Whenever  possible,  M&S  data  will  be  collected  in 
support  of  the  same  MOPs  for  which  live  data  was 
collected.  This  will  aid  the  V&V  process  by 
providing  common  data  points  for  comparison. 

3.5.3  Theater-Level  Analysis 
M&S  are  used  in  FoS  T&E  to  extend  assessments 
from  few-on-few  to  many-on-many  engagements  and 
large  theater  architectures.  M&S  are  the  only  tools 
that  can  be  used  to  examine  Theater-wide  FoS  issues 
in  scenarios  that  include  the  number  of  threats  that 
can  be  expected  under  tactical  conditions,  as  well  as 
providing  realistic  representations  of  DIA  approved 
threats.  This  method  can  be  used  to  examine  theater 
communications  networks  under  the  loading 
associated  with  projected  threat  levels  for  both  TBMs 
and  air  breathing  objects.  Examining  the  networks 
under  maximum  loading  provides  the  opportunity  to 
examine  effects  on  FoS  capabilities  resulting  from 
changes  in  the  timelines  for  critical  information  flow 
due  to  changes  in  message  update  rates,  as  well  as  the 
amount  of  information  the  system  receives. 

Theater-level  simulations  can  also  be  used  to 
compare  the  protection  effectiveness  of  the  FoS  as 
compared  to  the  same  systems  operating 
autonomously.  Once  timelines  for  the  transfer  of 
information  have  been  established  through  tests  and 
exercises,  the  change  in  effectiveness  of  attack 
operations  and  passive  defense  can  be  analyzed. 
Theater  level  simulations  provide  an  opportunity  to 
identify  potential  integration  or  interoperability 
issues  that  may  be  associated  with  the  deployment  of 
a  theater-wide  FoS.  Once  these  issues  have  been 
identified  and  analyzed,  system  and  test  requirements 
can  be  modified  to  address  the  problems. 
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Theater-level  analysis  will  be  accomplished,  in  part, 
by  the  use  of  simulations  to  extend  the  results  of  few- 
on-few  live  test  events  to  many-on-many  scenarios. 
The  primary  role  of  T&E  in  this  analysis  will  be  to 
provide  the  actual  test  data  to  aid  in  the  conduct  of 
FoS  performance  analysis  by  the  systems  engineer. 

3.6  Data  Feedback  Plan 

The  flow  of  data  in  the  evaluation  process  is  critical. 
Data  flow  needs  to  be  maintained  between  the 
various  M&S  tools  to  provide  inputs  and  feedback. 
Data  from  the  engineering  and  engagement  level 
models  will  be  used  as  inputs  and  validation  data  for 
the  mission  and  campaign  level  models.  Outputs 
from  the  mission  and  campaign-level  models  will  be 
used  in  the  planning  of  test  events,  extension  of  test 
results,  and  will  be  compared  with  the  data  from  the 
engineering  and  engagement  level  models  for 
validation  activities.  Data  from  live  test  events  will 
support  model  validation  activities  and  will  be  used 
as  feedback  to  the  model  developers.  This  general 
process  is  illustrated  in  Figure  4-7. 


Figure  3-7.  General  Data  Flow  Between  M&S  and 
Test  Events. 


Maintaining  the  exchange  of  data  and  information 
between  the  M&S  community  and  T&E  is  critical  to 
the  success  of  the  FoS  T&E  M&S  efforts.  According 
to  STEP,  feedback  from  tests  to  simulations  is  not 
only  required  for  VV&A,  but  also  to  allow 
progressive  improvement  in  the  M&S^  M&S 
developers  need  to  be  represented  in  the  test  planning 
process  so  that  they  may  gain  an  understanding  of  the 
event  and  to  identify  data  that  can  be  collected  for 
model  inputs  and  in  support  of  model  validation.  A 
matrix  of  common  data  between  the  various  models 
and  simulations  and  test  events  should  be  established 
to  aid  in  this  process. 

It  is  also  important  to  maintain  a  flow  of  data  and 
information  between  the  FoS  analysis  and  T&E 
efforts.  Analysis  results  may  uncover  problems  or 


shortcomings  with  the  models  that  may  need  to  be 
addressed  prior  to  their  use  for  T&E.  Results  from 
the  analysis  may  also  be  used  to  quantify  testing 
parameters  and  to  aid  in  the  identification  of  MOPs 
and  data  requirements  that  can  be  satisfied  with  the 
various  M&S  tools.  On  the  other  hand,  results  from 
T&E  events  may  reveal  areas  that  will  require  further 
analysis  or  uncover  requirements  that  may  need  to  be 
revised  or  added.  These  are  only  a  few  examples  of 
the  benefits  that  can  be  gained  from  the  interaction 
between  the  T&E  and  Analysis  functions.  In  order  to 
ensure  that  this  interaction  occurs,  representatives 
from  T&E  are  updated  on  the  progress  of  the  analysis 
and  often  participate  in  the  studies.  Likewise, 
members  from  the  analysis  team  will  assist  in  the 
execution  of  the  T&E  M&S  events.  This  close 
interaction  will  not  only  enhance  the  flow  of 
information  between  the  groups,  but  will  also  help  to 
leverage  valuable  resources. 

3.6.1  Model  And  Simulation  Data  Center 

All  test  and  M&S  output  data  needs  to  be  stored  and 
maintained  for  future  reference.  Data  from  FoS  T&E 
events  should  submitted  to  the  BMDO  Simulation 
Support  Center  (BMD  SSC),  which  is  part  of  the 
Defense  Modeling  and  Simulation  Office  (DMSO) 
Modeling  and  Simulation  Resource  Repository 
(MSRR).  The  BMD  SSC  is  a  data  archive,  access, 
management,  and  distribution  system  for  science  arid 
technical  data/information.  The  BMDSSC  M&S 
Repository  has  the  capability  to  store  and  provide 
access  to  TMD  and  NMD  test  and  exercise  data,  such 
as  SITS,  HWIL,  flight  tests;  or  to  provide  pointers  to 
where  that  data  is  stored. 

The  MSRR  is  a  collection  of  M&S  resources.  MSRR 
Resources  include  models,  simulations,  object 
models.  Conceptual  Models  of  the  Mission  Space 
(CMMS),  algorithms,  instance  databases,  data  sets, 
data  standardization  and  administration  products, 
documents,  tools  and  utilities. 

Both  of  these  programs  are  critical  to  the  successful 
implementation  of  M&S  in  any  large  scale  effort  as 
they  provide  valuable  M&S  resources  and  maintain 
data  that  can  be  used  for  V&V,  model  input,  or  future 
reference. 

3.6.2  Data  Commonality  Matrices 

The  purpose  of  the  Data  Commonality  Matrices 
would  be  to  identify  common  data  requirements 
between  FoS  T&E  events  and  the  M&S  tools.  Data 
obtained  from  T&E  events  could  be  used  either  as 
input  for  M&S  or  as  data  for  verification/validation 
activities.  In  order  for  this  process  to  occur,  the  test 
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events  that  can  provide  relevant  M&S  data  must  be 
identified  and  the  relevant  data  extracted. 

The  Data  Commonality  Matrices  would  list  each  type 
of  FoS  T&E  event  and  the  FoS  data  that  is  expected 
from  the  event.  The  data,  as  it  relates  to  FoS  test 
objectives  and  requirements,  will  be  matched  with 
the  M&S  level  that  might  benefit  from  that  data. 
Additional  data  may  also  be  obtained  from  these  tests 
if  proper  planning  is  used  to  identify  the  useful  data 
and  prepare  for  the  data  collection. 

The  FoS  T&E  events  offer  a  vast  amount  of  data  for 
use  in  BMDO  models  and  simulations.  This  data  can 
be  used  as  either  input  for  routine  simulation  runs, 
such  as  the  scenario  data  obtained  from  the  CINC 
Assessments,  or  for  model  verification/validation, 
such  as  the  system  data  obtained  from  SITs  or 
MDAP  tests.  Although  most  of  the  FoS  test  data 
primarily  benefits  the  Engineering  or  Engagement 
level  models  and  the  EADTB  SSRs,  some  data  may 
also  be  used  by  the  Theater  level  models  either 
directly  or  indirectly.  Once  test  data  has  been  used  to 
verify  a  system  in  an  Engineering  or  Engagement 
level  model,  that  information  can  then  be  used  to 
validate  the  Theater  level  model’s  representation  of 
that  system. 

Through  the  use  of  these  matrices,  the  model 
developers  would  have  an  idea  of  which  T&E  events 
are  likely  to  provide  data  of  interest  to  their  specific 
model  or  purpose.  The  matrices  would  also  aid  the 
T&E  team  in  selecting  models  to  address  specific  test 
objectives. 

4.  CONCLUSIONS 

M&S  will  be  an  important  tool  in  conducting  FoS 
interoperability  assessments.  Live  tests  are  normally 
conducted  when  conditions  are  perfect  and  ensures 
the  most  successful  outcome  possible,  M&S  tools 
will  allow  for  the  testing  of  the  FoS  when  conditions 
are  less  favorable  and  more  representative  of  actual 
war  time  situations.  Using  M&S  with  the  actual 
tactical  hardware  provides  higher  confidence  that 
these  systems  when  deployed  to  the  theater  will 
coexist  in  an  interoperable  environment  among  all 
Service  weapons  and  C2  systems.  Future  tests  will 
be  conducted  on  an  event-driven  basis  as  weapon 
system  and  C2  center  capability  becomes  available 
and  requires  testing  to  ensure  requirements  can  be 
met. 

It  is  important  that  each  test  continue  to  provide 
information  that  will  enhance  future  tests.  Whether  it 
is  using  live  data  to  increase  the  fidelity  of  M&S  or 
tools  predicting  live  test  objectives,  M&S  will  play  a 
vital  role  in  making  sure  that  the  FoS  is  up  to  the 


standards  necessary  to  defend  any  asset  anywhere  in 
the  world  when  the  time  comes.  It  is  also  very 
important  for  BMDO  to  continue  the  development  of 
a  core  set  of  models  and  simulations  and  to  support 
the  development,  enhancement  and  validation  of 
those  core  M&S  tools 

The  Implementation  of  M&S  into  the  T&E  program 
will  provide  the  opportunity  to  conduct  tests  and 
verify  requirements  that  would  otherwise  be  too 
costly  or  difficult  to  accomplish.  Identifying  and 
documenting  procedures  from  STEP  will  ensure  the 
“model-test-model”  paradigm  will  lead  to  the 
successful  integration  of  independently  developed 
TMD  weapon  systems  and  C2  centers  into  a  cohesive 
FoS  prior  to  deployment  of  the  systems.  It  is  also 
important  that  the  T&E,  M&S,  and  analysis 
communities  work  closely  together  to  interchange 
and  compliment  data  to  support  M&S  development, 
enhancement  and  validation.  This  relationship  would 
also  enable  more  effective  use  of  M&S  throughout 
the  development  and  testing  of  the  system. 
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